{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLP实现mnist分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 逻辑回归模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用于解决分类问题的一种模型。根据数据特征或属性，计算其归属于某一类别的概率P(x)，根据概率数值判断其所属类别。\n",
    "- 主要应用场景：二分类问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 逻辑回归数学表达式\n",
    "\n",
    "![logistic_regression formula](images/06_logistic_regression_formula.png)\n",
    "\n",
    "其中，y为类别结果，P为概率，x为特征值，a、b为常量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型评估回顾\n",
    "\n",
    "- 目的：通过模型评估对比模型表现、确定合适的模型参数（组）\n",
    "- 方法：计算测试数据集预测准确率以评估模型表现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 皮马印第安人糖尿病数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于数据集中包括的某些诊断测量来诊断性地预测患者是否患有糖尿病\n",
    "\n",
    "输入变量包括：独立变量包括患者的怀孕次数，葡萄糖量，血压，皮褶厚度，体重指数，胰岛素水平，糖尿病谱系功能，年龄\n",
    "\n",
    "输出结果：是否患有糖尿病\n",
    "\n",
    "数据来源：[Pima Indians Diabetes dataset](https://www.kaggle.com/uciml/pima-indians-diabetes-database)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**任务:** 通过怀孕次数、胰岛素水平、体重指数、年龄四个特征预测是否患有糖尿病"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\niubi\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "#load mnist data from keras\n",
    "from keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "(feature_train, label_train), (feature_test, label_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAELCAYAAAB3QSUaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYFNXZ9/HvjQiKyBqjokFUYkAUXACXIGCCG+KCRtSgCC74aEQxiUtcCEYUl+gTxAXjhtsrMTGCGHmVRBQXXBPyCogLRAQHBER2hADn/aPnTE0PPcM003Wqeub3ua6+6K6qrjo9c3Pm7lNnMeccIiISr3pJF0BEpC5QZSsiEoAqWxGRAFTZiogEoMpWRCQAVbYiIgEEr2zNbIyZ3VjoY6W4KS4kl9oUF1bIfrZm9gWwK7AR2ATMAp4A/uic21zDc/cEnnLO7ZnHe4YD1wPry23u6JybW5OySH5SGBcG3AZcWLrpEeAap07nQaUtLsq9twHw/4DG2/L+ysSR2Z7knNsZ2ItMQF9DJpiT8ifnXONyD1W0yUhTXAwGTgU6AR2BPsDFCZWlrktTXHhXAYsLfdLYmhGccyuccy8AZwLnmdkBAGY21sxG+OPM7GozW2hmJWZ2oZk5M2tb/lgz2wmYBLQys9Wlj1ZxlV3ik5K4OA+4yzm3wDn3FXAXMLDAH1XykJK4wMz2Bs4BRhb6M8beZuucew9YABxVcZ+ZHQ/8EugFtAV6VHKONcAJQEm5DLXEzLqZ2fKtFOEkM1tmZjPN7JIafRgpmITjogPw73Kv/126TRKWgvpiNHAdsK4GHyOnUDfISoAWObb3Ax5zzs10zq0FbsrnpM65N51zzao45FmgPbALcBEwzMzOzucaEquk4qIxsKLc6xVA49K2XEleInFhZn2B+s655/MqbTWFqmz3AJbl2N4KmF/u9fwcx2wz59ws51yJc26Tc+5tYBTws0JeQ2okkbgAVgNNyr1uAqzWDbLUCB4XpU0PdwBDCnXOimKvbM2sC5kf3ps5di8Eyt/t+0EVpyrEfwQHKHtJgYTjYiaZm2Nep9JtkrAE4+KHQBvgDTNbBPwV2N3MFplZmzzPlVNsla2ZNTGzPsA4Ml0wPspx2LPAIDNrb2aNgGFVnPJroKWZNc2jDKeYWXPL6ApcDkzI42NIgaUhLsh0L/qlme1ReuPkV8DYPN4vBZaCuJhBpvI+qPRxYek5DqJAGXQcle1EM1tFpoDXA3cDg3Id6JybBNwDTAE+B6aV7lqf49jZwDPAXDNbbmatzOwoM1tdRVnOKj3vKjL/wW53zj2+bR9LaihNcfEgMBH4iMx/sr+VbpPwUhEXzrmNzrlF/kGmGWNz6etNNfyMQIEHNdSUmbUnE/wNnXMbky6PpIPiQnIptrhIfG4EM+trZg3MrDlwOzCxGH5wEi/FheRSzHGReGVLZuTOEmAOmSF76gsroLiQ3Io2LlLVjCAiUlulIbMVEan1VNmKiARQP5+DzaxOtDk45zTwIQ91JS6Apc65XZIuRLFQXGRTZitSffOSLoCkUrXiQpWtiEgAqmxFRAJQZSsiEoAqWxGRAFTZiogEoMpWRCSAvPrZiqTNoYceCsBll10GwIABAwB44oknABg9ejQA//znPxMonUhEma2ISAB5TUQTYkTIdtttB0DTprknWPcZTKNGjQD40Y9+BMAvfvELAH7/+98DcPbZ0bqO3333HQC33XYbADfdVPU6cRpBlp8kRgoddNBBALz66qsANGnSJOdxK1Zk1nVs2bJlIS77oXOucyFOVBcUwwiyn/70pwA8/fTTZdt69Mgs2vvJJ59U9zTVigtltiIiAQRvs23dujUADRo0AODII48EoFu3bgA0a5ZZafj000+v1vkWLFgAwD333ANA3759AVi1alXZMf/+978BeP3112tUdkle165dAXjuueeA6BuQ/4bmf+8bNmwAooz28MMPB7Lbbv0xkozu3bsD0e/o+edjWUG8Sl26dAHg/fffj/1aymxFRAIIktn69jWI2tgqa5Otrs2bNwNwww03ALB6dWYdN9/2snDhwrJjv/32WyCvNhhJCd82f8ghhwDw1FNPAbD77rvnPP6zzz4D4I477gBg3LhxALz11ltAFC8AI0eOjKHEUl09e/YE4Ic//CEQNrOtVy+TZ+69994A7LXXXmX7zOK5ZaPMVkQkAFW2IiIBBGlG+PLLL8uef/PNN0D1mxHeffddAJYvXw7A0UcfDUQ3N5588smClVPS58EHHwSyu/JVxTc3NG7cGIhuivqvrB07dixwCWVb+QEo06ZNC35t3wx10UUXAVHzFMDs2bNjuaYyWxGRAIJktsuWLSt7ftVVVwHQp08fAP71r38BUdctb/r06QAcc8wxAKxZswaADh06AHDFFVfEWGJJmh+Ge+KJJwJb3rTwGevEiROBaDBLSUkJEMWVvzn6k5/8JOd5JDn+JlUSHn744azX/sZqnJTZiogEEHxQw/jx44GoC5jvhN6pUycALrjgAiDKVHxG682cOROAwYMHx19YCc53E5w8eTIQDcP1gxYmTZoERG24fmil79LlM5YlS5YA0YAW31XQZ8oQte9qkpqwfLv5rrvumlgZKt4z8vEWJ2W2IiIBJDbF4sqVK7Ne+wlDPH+X8E9/+hMQZSZSO+23335A1KbvM4+lS5cC0SCVxx9/HIgGsfztb3/L+ndrdtxxx7Lnv/rVrwDo379/jcou+enduzeQ/bsIxWfTfjCD99VXX8V+bWW2IiIBpGby8OHDhwPRXWjfFterVy8AXnnllUTKJfFp2LBh2XPfRu+zHt+W7/tifvDBB0BhsyE/KZKE5adF9fx9mBB8nPkM99NPPwWyJ66KizJbEZEAUpPZ+l4Hvq3W3yF+6KGHAJgyZQoQZTj33XcfEN2lluJz8MEHlz33Ga13yimnAJoWsy6IY3pD34vl+OOPB+Ccc84B4Nhjj8067uabbwaiEapxUmYrIhJAajJbb86cOQAMHDgQgMceewyAc889N+vfnXbaCYgW9is/paIUh7vvvrvsuR/Z5TPZQme0frSSerWkT4sWLbZ6jO+H7+PE38vZc889gWgxAt+zxP++161bB0RzrKxfvx6A+vUzVd+HH35Y8w9QTcpsRUQCSF1m6/mJhP2YZZ8F+QXabr31ViCa9PeWW24BwvSXk5rx82KUn1Tet72/8MILsVzTZ7Tl2/j9/BsSls82/e9izJgxAFx33XWVvsePOvOZ7caNGwFYu3YtALNmzQLg0UcfBaJ7O/4b0tdffw1Ey2j5Xi1xzfCVizJbEZEAUpvZejNmzACgX79+AJx00klA1JZ78cUXA9HSGn6WMEkvn1X4djaAxYsXA9GIwZryfXh9/23Pz8kB8Jvf/KYg15L8XHrppQDMmzcPiBZ9rYqfE9vPrfLxxx8D8M4771Trmn4ulV122QWAuXPn5lHiwlBmKyISQOozW8/3g/MrM/jZnfxdRb8ssp+R/7XXXgtbQKkRf5e4pr1KfEbrZwHzcy34trq77rqr7Fg/v4Ik4/bbbw92LX+vx3vuueeCXdtTZisiEkDqM1t/F/JnP/sZAF26dAGijNbzdyOnTp0asHRSKDXtheB7NvhM9swzzwRgwoQJAJx++uk1Or/ULiGXTfeU2YqIBJC6zNbPCHTZZZcBcNpppwGw22675Tx+06ZNQNTWpxFC6ef7SpZfD+zUU08F8l9b7sorrwTgxhtvBKJ5cJ9++mkgmjVMJGnKbEVEAkg8s/UZq19Tyme0bdq0qfJ9foSIHzkW18gjKTw/cqj8aC4fB36VZT8S6JtvvgHg8MMPB6K5MfxYeT823vfDfPnllwG4//774/sAUrT8tym/Mkh1++kWgjJbEZEAgme2fob0/fffH4B7770XgHbt2lX5Pj9rz5133glEd5nVRls7bLfddkA0usj3HvBr1fkRghW9/fbbQDTf8bBhw2ItpxQ3/23KzwoWkjJbEZEAVNmKiAQQazOCnxT4wQcfLNvmO5/vs88+Vb7Xfz30wyv9jQ8/PZsUr2nTpgHZy6H4wSqev2Hmm508f8Ns3LhxQP5dxUQAjjjiCADGjh0b7JrKbEVEAihoZnvYYYcB0ZDJrl27ArDHHnts9b1+EmDf9cdPDu4XgpTaw08K4wesQDRVpp9ApqJRo0YB8MADDwDw+eefx1lEqaXKD6QJTZmtiEgABc1s+/btm/VvLn7CmBdffBGIlrfwbbMhlhSWdCg/naKf5LviZN8ihTBp0iQAzjjjjMTKoMxWRCQAKz9kcqsHm1X/4CLmnEuuYacI1ZW4AD50znVOuhDFQnGRTZmtiEgAqmxFRAJQZSsiEoAqWxGRAFTZiogEkG8/26XAvDgKkiJ7JV2AIlQX4gIUG/lSXJSTV9cvERHZNmpGEBEJQJWtiEgAqmxFRAJQZSsiEoAqWxGRAFTZiogEoMpWRCQAVbYiIgGoshURCUCVrYhIAKpsRUQCUGUrIhJA8MrWzMaY2Y2FPlaKm+JCcqlVceGcK9gD+AJYB6wClgNvA/8D1CvAuXsCC/J8z9HAFGAF8EUhP6seRR0XzYDHgcWlj+FJ/4zq4iOFcXEVMKO0PP8Brirk540jsz3JObczmTkebwOuAR6J4TrVsQZ4lMwPUZKVprj4X6AR0AboCpxrZoMSKktdl6a4MGAA0Bw4HrjMzM4q2Nlj+EvVq8K2rsBm4IDS12OBEeX2Xw0sBEqACwEHtC1/LLATmb+Am4HVpY9WeZSrF8psE3ukLS7ITGrdpdzr64A3kv451bVH2uIiR/nuAUYX6vPG3mbrnHsPWAAcVXGfmR0P/JJMZdgW6FHJOdYAJwAlzrnGpY8SM+tmZsvjK73EJQVxYRWeH7ANH0MKLAVx4a9lpWWYuW2fZEuhbpCVAC1ybO8HPOacm+mcWwvclM9JnXNvOueaFaKAkoik4uL/Atea2c5m1hY4n0yzgqRDGuqL4WTqx8fyuUZVQlW2ewDLcmxvBcwv93p+jmOk9koqLi4n8zXzM2AC8AyZbErSIdH6wswuI9N2e6Jzbn2hzht7ZWtmXcj88N7MsXshsGe51z+o4lRaLK0WSTIunHPLnHP9nXO7Oec6kPl/8F6+55HCS7q+MLPzgWuBnzrnCvoHOLbK1syamFkfYBzwlHPuoxyHPQsMMrP2ZtYIGFbFKb8GWppZ0zzKUM/MdgC2z7y0HcysQR4fQwosJXGxr5m1NLPtzOwEYDCZGyuSkJTERX/gVuAY59zcPIpfLXFUthPNbBWZFP964G4gZ7ca59wkMnf8pgCfA9NKd22RujvnZpP5ujfXzJabWSszO8rMVldRlu5kvi6+BLQuff7KNn0qqak0xcWhwEdk+lOOBPo75wp2I0Tykqa4GAG0BN43s9WljzHb+sEqStVS5mbWnkyn4obOuY1Jl0fSQXEhuRRbXCQ+N4KZ9TWzBmbWHLgdmFgMPziJl+JCcinmuEi8sgUuBpYAc4BNwCXJFkdSQnEhuRRtXKSqGUFEpLZKQ2YrIlLrqbIVEQmgfj4Hm1mdaHNwztnWjxKvrsQFsNQ5t0vShSgWiotsymxFqm9e0gWQVKpWXKiyFREJQJWtiEgAqmxFRAJQZSsiEoAqWxGRAFTZiogEoMpWRCSAvAY1pNENN9wAwE03ZZYjqlcv8/ejZ8+eZce8/vrrwcslIsnZeeedAWjcuDEAJ554IgC77JIZe3D33XcDsH59wVa92SpltiIiARRtZjtw4EAArrnmGgA2b96ctV+zmYnUHW3atAGi+uCII44A4IADcq9Qv/vuuwNw+eWXx1+4UspsRUQCKNrMdq+99gJghx12SLgkEsJhhx0GwDnnnANAjx49AOjQoUPWcb/+9a8BKCkpAaBbt24APPXUUwC8++678RdWYteuXTsAhg4dCkD//v0B2HHHHQEwy8wlNX9+ZrXzVatWAdC+fXsA+vXrB8D9998PwOzZs2MvszJbEZEAVNmKiARQdM0IvXr1AmDIkCFZ2/3XgD59+gDw9ddfhy2YxOLMM88EYNSoUQB873vfA6Kvia+99hoQdem58847s97vj/P7zzrrrHgLLLFo2rQpALfffjsQxYXv4lXRZ599BsBxxx0HwPbbbw9E9YSPI/9vCMpsRUQCKJrM1t/oeOyxx4DoL53nM5p58zS/czGrXz8Tkp07dwbgoYceAqBRo0YATJ06FYCbb74ZgDfffBOAhg0bAvDss88CcOyxx2ad94MPPoiz2BKzvn37AnDhhRdWedycOXMAOOaYY4DoBlnbtm1jLF31KLMVEQmgaDLb8847D4BWrVplbfdtdk888UToIkkMfNeuhx9+OGv75MmTgaitbuXKlVn7/faKGe2CBQsAePzxxwtfWAnmjDPOyLn9iy++AOD9998HokENPqP1fJevJCmzFREJIPWZrb9beP755wPRsNzly5cDMGLEiGQKJgXl22Cvu+46IBpu7Tud+wmHKma03vXXX59zux+OuWTJksIVVoK76KKLABg8eDAAr7zyCgCff/45AIsXL67y/bvuumuMpaseZbYiIgGkNrP1E0s899xzOfePHj0agClTpoQqkhTYsGHDyp77jHbDhg0AvPzyy0DUBrdu3bqs9/ph2r6NtnXr1kDUr9Z/45kwYUIsZZew/PDr4cOHb9P7/cQ0SVJmKyISQGoz2+OPPx6Ajh07Zm3/xz/+AUQjiqT4NGvWDIBLL720bJtvo/UZ7amnnprzvb6/5NNPPw3AoYcemrX/L3/5CwB33HFHAUssaefb5nfaaaec+w888MCs12+//TYA06ZNi7dg5SizFREJIHWZrc9obrvttqztfqSQ72+7YsWKsAWTgmnQoAGQe1y6z1C+//3vAzBo0CAATj75ZCCaDNovd+IzYv+vn0pxzZo1sZRdkuVHEu6///4A/Pa3vwWgd+/eWcf55bEqLirg2359XG3atCm+wlagzFZEJIDUZLZb630wd+5cQLN51Qa+x0H5vq9+Vq7//Oc/QOXLGvnMxPe39cubLF26FICJEyfGUGJJip+t6+CDDwai+sH/3n0vFR8Xvg3W3/PxmbDn59447bTTgOjej4/JOCmzFREJIDWZbWULN3oV23ClePnRf+V7HLz44osAtGjRAohmb/L9ZMeOHQvAsmXLABg3bhwQZTj+tRQ/36YPUYb617/+NeuYm266CYBXX30VgLfeeguI4sdvr7jgo/8GNXLkSAC+/PJLAMaPH192TFzLmyuzFREJIPHM9qCDDgK2nK3J85nNJ598EqxMEkb5xRd9xrE13bt3B6IFH/03Id+mL8XLt8/6rBXgqquuyjpm0qRJQDSC1H9L8vHz0ksvAVG/Wt8W6/td+0z3lFNOAaL+2n//+9/LruFXg/j222+zrj19+vRt/GQZymxFRAJIPLP1s/c0b948a/s777wDwMCBA0MXSVLML1XtM1rfa0FttsVru+22A6KZ3/xy9BD1l7722muB6PfsM1q/ose9994LRL0W/Bpkl1xyCRDNodKkSRMAjjzySCBaAt3344Zo7mTPz4279957b/NnBGW2IiJBWGX9GXMebFb9g6vJj+Co2AthwIABADzzzDOFvuRWOecs+EWLWBxxsTU+bnz8+l4JMc9b+6FzrnOcF6hNqhsXPvv07bBr164t21dx/trDDjsMiEaAnXDCCUD0jed3v/sdEK1VWHHFhsqcffbZZc9//vOfZ+278sorgWju3ByqFRfKbEVEAkgss/V/eXybbMXMdp999gGSWS1XmW1+Qma2xx13HBDddVZmm17VjYuFCxcCUY+C8v1cZ8+eDUSzeVW2Sq6f59b3nw055wHKbEVE0iN4bwTfr7ZXr15AlNH6/nD33XcfoDkQJDf/jUdqj0WLFgFRZtuwYcOyfZ06dco61n+jmTp1KhCN/PKr7AbOaPOizFZEJABVtiIiAQRvRvBLouy2225Z27/66isgu0OzSEVvvPEGUPnk0FJ8/BBsPzHRIYccUrbPL1H+6KOPAtEQ2hBTIhaaMlsRkQASH64rko8ZM2YA0XBMf8Ns3333BWLv+iUxWLVqFQBPPvlk1r+1jTJbEZEAgme2vpOyX0q4W7duoYsgtcCtt94KwMMPPwzALbfcAsCQIUMAmDVrVjIFE6mEMlsRkQASn4gmjTRcNz9JxIWfKu/ZZ58FokEyfvkUP1FJgZc013DdPNSV+gIN1xURSQ9ltjkos81PknHhM1zfZuun6+vYsSNQ8LZbZbZ5qCv1BcpsRUTSQ5ltDsps81NX4gJltnlRXGRTZisiEkC+/WyXAuFn8w5rr6QLUITqQlyAYiNfioty8mpGEBGRbaNmBBGRAFTZiogEoMpWRCQAVbYiIgGoshURCUCVrYhIAKpsRUQCUGUrIhKAKlsRkQBU2YqIBKDKVkQkAFW2IiIBqLIVEQkgeGVrZmPM7MZCHyvFTXEhudSquHDOFewBfAGsA1YBy4G3gf8B6hXg3D2BBXm+52hgCrAC+KKQn1WPoo6LocBcYCVQAvwvUD/pn1Nde6QwLmKtL+LIbE9yzu1MZkLd24BrgEdiuE51rAEeBa5K6PoSSVNcTAQOcc41AQ4AOgGXJ1SWui5NcRFvfRHDX6peFbZ1BTYDB5S+HguMKLf/amAhmQzjQsABbcsfC+xE5i/gZmB16aNVHuXqhTLbxB5pjYvSc7UE/g7cn/TPqa490hoXcdUXsbfZOufeAxYAR1XcZ2bHA78s/XBtgR6VnGMNcAJQ4pxrXPooMbNuZrY8vtJLXJKOCzP7uZmtJLN0SyfgwRp9ICmIpOMiTqFukJUALXJs7wc85pyb6ZxbC9yUz0mdc28655oVooCSiMTiwjn3f1ymGWE/YAzwdT7XkFjVyvoiVGW7B7Asx/ZWwPxyr+fnOEZqr8Tjwjn3GTATuD+ua0jeEo+LOMRe2ZpZFzI/vDdz7F4I7Fnu9Q+qOJVWpqxFUhYX9YF9C3AeqaGUxUVBxVbZmlkTM+sDjAOecs59lOOwZ4FBZtbezBoBw6o45ddASzNrmkcZ6pnZDsD2mZe2g5k1yONjSIGlJC4uNLPvlz7fH/gN8I9qfwgpuJTERaz1RRyV7UQzW0Umxb8euBsYlOtA59wk4B4yfds+B6aV7lqf49jZwDPAXDNbbmatzOwoM1tdRVm6k7kr+RLQuvT5K9v0qaSm0hQXPwY+MrM1ZGLjJeC6bftYUkNpiotY6wsr7eqQCmbWHpgBNHTObUy6PJIOigvJpdjiIvG5Ecysr5k1MLPmwO3AxGL4wUm8FBeSSzHHReKVLXAxsASYA2wCLkm2OJISigvJpWjjIlXNCCIitVUaMlsRkVpPla2ISAD18znYzOpEm4NzzpIuQzGpK3EBLHXO7ZJ0IYqF4iKbMluR6puXdAEklaoVF6psRUQCUGUrIhKAKlsRkQBU2YqIBKDKVkQkgLy6foUwatQoAC6/PLP+3owZMwDo06cPAPPm6YawiBQfZbYiIgGkJrNt06YNAOeccw4AmzdvBqB9+/YAtGvXDlBmW9fst99+AGy//fYAdO/eHYD778+sYuPjZGsmTJgAwFlnnVW2bcOGDQUrpyTDx8WRRx4JwK233grAj3/848TKVBlltiIiAaQms12yZAkAU6dOBeDkk09OsjiSkA4dOgAwcOBAAM444wwA6tXL5AWtWrUCooy2urPW+XgaM2ZM2bahQ4cCsHLlyhqWWpLStGlm1ZspU6YAsGjRIgB22223rNdpoMxWRCSA1GS2a9asAdQmW9eNHDkSgN69e8dy/gEDBpQ9f+SRRwB46623YrmWhOczWmW2IiJ1lCpbEZEAUtOM0KxZMwA6deqUcEkkSZMnTwa2bEZYvHgxEH319zfMKnb98l2AevToEWs5JZ3M0jsVtTJbEZEAUpPZNmrUCIDWrVvn3N+lSxcAZs+eDehGWm31wAMPADB+/Pis7f/973+Brd/waNKkCRAN8/Zdxbzy5/3ggw9qVlhJHd8VcIcddki4JFtSZisiEkBqMtuSkhIAxo4dC8Dw4cOz9vvXy5cvB+Dee+8NVTQJaOPGjQDMnz9/m95/3HHHAdC8efOc+xcsWFD2fP369dt0DUm/zp07A/DOO+8kXJKIMlsRkQBSk9l6N998M7BlZitSFT/BzEUXXQTAjjvumPO4YcOGBSuTxM9/E1qxYgUQDd/dd999EytTZZTZiogEkLrM1qusH6UIQP/+/QG49tprAWjbti0QTblX0fTp04GoV4PUDv4ezhtvvAFEiwykkTJbEZEAUpvZ5juFntQOfhL5c889F4BevXrlPK5bt25A5fHhp030me9LL70EwLp16wpWVpF8KLMVEQkgtZmt1C0HHHAAAC+88AJQ+UjC6vJteH/84x9rVjApSi1btky6CFtQZisiEoAyW0kVP2vT1mZv2lpvFX9X+oQTTgBg0qRJhSqiFIE0LqulzFZEJIDUZraVZS5+KWvNjVC7+Fm6evbsCURL2r/88ssAfPfdd1W+/4ILLgBgyJAhMZVQ0swv+Kh+tiIidZzl04/VzIJ1et20aRNQeT/Kjh07AjBr1qyCX9s5l97p3lMoZFxUxo+J/+abb7K2n3TSSUDB2mw/dM51LsSJ6oKQcXH66acD8Oc//xmI+lPvv//+QOzzX1crLpTZiogEkNo22zFjxgBw8cUX59w/ePBgAIYOHRqsTJJefh5bqZv87F+e783SsGHDJIqTkzJbEZEAUpvZ+rXGpPbxM3Mde+yxZdteffVVIP+5CwYNGgTAqFGjClQ6KUYTJkwAonqjXbt2QPTN99JLL02mYOUosxURCSC1vRG8Tz/9FNhy5nXfD9fPYzpnzpyCXVO9EfJT3bjwM3Vdf/31ABxzzDFl+/bee29g62uPtWjRAoDevXsDMHr0aAB23nnnrON8huxHEvl+mDWk3gh5SKK++MMf/gBE33h23XVXYOv9tGtIvRFERNIitW223syZMwHYZ599srZrBYfi40f9+Rm+yrv66qsBWLVqVZXn8NnwIYccAmzZD/u1114D4IEHHgAKltFKkfFxsWHDhoRLElFmKyISgCpbEZEAUt+M4Cd/9sMupXa65JJLtul9ixcvBmDixIkAXHFBm1Z5AAABLklEQVTFFUDsN0Qk5Zo0aQLAKaecAsDzzz+fZHEAZbYiIkGkPrP1E818/PHHALRv3z7J4kgNDBw4EIimQTzvvPOq/V7ftW/t2rXAlsve+CkapW7r168fAOvXrweieiMNlNmKiASQ+szWT4124IEHJlwSqanp06cD0dDJ9957r2zfiBEjAGjevDkA48ePB2Dy5MlANBxz0aJFYQorRWnq1KlA9A04TUvXK7MVEQkg9cN1k6DhuvmpK3GBhuvmRXGRTZmtiEgAqmxFRAJQZSsiEoAqWxGRAFTZiogEkG8/26VArGsCp8BeSRegCNWFuADFRr4UF+Xk1fVLRES2jZoRREQCUGUrIhKAKlsRkQBU2YqIBKDKVkQkAFW2IiIBqLIVEQlAla2ISACqbEVEAvj/e+ldTPMsLg4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "for i in range(9):\n",
    "  plt.subplot(3,3,i+1)\n",
    "  plt.tight_layout()\n",
    "  plt.imshow(feature_train[i], cmap='gray', interpolation='none')\n",
    "  plt.title(\"Digit: {}\".format(label_train[i]))\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(feature_train.shape,label_train.shape,feature_test.shape,label_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'> <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "print(type(feature_train),type(label_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   3  18  18  18 126 136\n",
      "  175  26 166 255 247 127   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  30  36  94 154 170 253 253 253 253 253\n",
      "  225 172 253 242 195  64   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0  49 238 253 253 253 253 253 253 253 253 251\n",
      "   93  82  82  56  39   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0  18 219 253 253 253 253 253 198 182 247 241\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  80 156 107 253 253 205  11   0  43 154\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0  14   1 154 253  90   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0 139 253 190   2   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0  11 190 253  70   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0  35 241 225 160 108   1\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0  81 240 253 253 119\n",
      "   25   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0  45 186 253 253\n",
      "  150  27   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  16  93 252\n",
      "  253 187   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 249\n",
      "  253 249  64   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0  46 130 183 253\n",
      "  253 207   2   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0  39 148 229 253 253 253\n",
      "  250 182   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0  24 114 221 253 253 253 253 201\n",
      "   78   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  23  66 213 253 253 253 253 198  81   2\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0  18 171 219 253 253 253 253 195  80   9   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0  55 172 226 253 253 253 253 244 133  11   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0 136 253 253 253 212 135 132  16   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]]\n"
     ]
    }
   ],
   "source": [
    "print(feature_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x26075053f60>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADgpJREFUeJzt3X+MVfWZx/HPs1j+kKI4aQRCYSnEYJW4082IjSWrxkzVDQZHrekkJjQapn8wiU02ZA3/VNNgyCrslmiamaZYSFpKE3VB0iw0otLGZuKIWC0srTFsO3IDNTjywx9kmGf/mEMzxbnfe+fec++5zPN+JeT+eM6558kNnznn3O+592vuLgDx/EPRDQAoBuEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxDUZc3cmJlxOSHQYO5u1SxX157fzO40syNm9q6ZPVrPawFoLqv12n4zmybpj5I6JQ1Jel1St7sfSqzDnh9osGbs+ZdJetfd33P3c5J+IWllHa8HoInqCf88SX8Z93goe+7vmFmPmQ2a2WAd2wKQs3o+8Jvo0OJzh/Xu3i+pX+KwH2gl9ez5hyTNH/f4y5KO1dcOgGapJ/yvS7rGzL5iZtMlfVvSrnzaAtBoNR/2u/uImfVK2iNpmqQt7v6H3DoD0FA1D/XVtDHO+YGGa8pFPgAuXYQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVfMU3ZJkZkclnZZ0XtKIu3fk0RTyM23atGT9yiuvbOj2e3t7y9Yuv/zy5LpLlixJ1tesWZOsP/XUU2Vr3d3dyXU//fTTZH3Dhg3J+uOPP56st4K6wp+5zd0/yOF1ADQRh/1AUPWG3yXtNbM3zKwnj4YANEe9h/3fcPdjZna1pF+b2f+6+/7xC2R/FPjDALSYuvb87n4suz0h6QVJyyZYpt/dO/gwEGgtNYffzGaY2cwL9yV9U9I7eTUGoLHqOeyfLekFM7vwOj939//JpSsADVdz+N39PUn/lGMvU9aCBQuS9enTpyfrN998c7K+fPnysrVZs2Yl173vvvuS9SINDQ0l65s3b07Wu7q6ytZOnz6dXPett95K1l999dVk/VLAUB8QFOEHgiL8QFCEHwiK8ANBEX4gKHP35m3MrHkba6L29vZkfd++fcl6o79W26pGR0eT9YceeihZP3PmTM3bLpVKyfqHH36YrB85cqTmbTeau1s1y7HnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOfPQVtbW7I+MDCQrC9atCjPdnJVqffh4eFk/bbbbitbO3fuXHLdqNc/1ItxfgBJhB8IivADQRF+ICjCDwRF+IGgCD8QVB6z9IZ38uTJZH3t2rXJ+ooVK5L1N998M1mv9BPWKQcPHkzWOzs7k/WzZ88m69dff33Z2iOPPJJcF43Fnh8IivADQRF+ICjCDwRF+IGgCD8QFOEHgqr4fX4z2yJphaQT7r40e65N0g5JCyUdlfSAu6d/6FxT9/v89briiiuS9UrTSff19ZWtPfzww8l1H3zwwWR9+/btyTpaT57f5/+ppDsveu5RSS+5+zWSXsoeA7iEVAy/u++XdPElbCslbc3ub5V0T859AWiwWs/5Z7t7SZKy26vzawlAMzT82n4z65HU0+jtAJicWvf8x81sriRltyfKLeju/e7e4e4dNW4LQAPUGv5dklZl91dJ2plPOwCapWL4zWy7pN9JWmJmQ2b2sKQNkjrN7E+SOrPHAC4hFc/53b27TOn2nHsJ69SpU3Wt/9FHH9W87urVq5P1HTt2JOujo6M1bxvF4go/ICjCDwRF+IGgCD8QFOEHgiL8QFBM0T0FzJgxo2ztxRdfTK57yy23JOt33XVXsr53795kHc3HFN0Akgg/EBThB4Ii/EBQhB8IivADQRF+ICjG+ae4xYsXJ+sHDhxI1oeHh5P1l19+OVkfHBwsW3vmmWeS6zbz/+ZUwjg/gCTCDwRF+IGgCD8QFOEHgiL8QFCEHwiKcf7gurq6kvVnn302WZ85c2bN2163bl2yvm3btmS9VCrVvO2pjHF+AEmEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxXF+M9siaYWkE+6+NHvuMUmrJf01W2ydu/+q4sYY57/kLF26NFnftGlTsn777bXP5N7X15esr1+/Pll///33a972pSzPcf6fSrpzguf/093bs38Vgw+gtVQMv7vvl3SyCb0AaKJ6zvl7zez3ZrbFzK7KrSMATVFr+H8kabGkdkklSRvLLWhmPWY2aGblf8wNQNPVFH53P+7u5919VNKPJS1LLNvv7h3u3lFrkwDyV1P4zWzuuIddkt7Jpx0AzXJZpQXMbLukWyV9ycyGJH1f0q1m1i7JJR2V9N0G9gigAfg+P+oya9asZP3uu+8uW6v0WwFm6eHqffv2JeudnZ3J+lTF9/kBJBF+ICjCDwRF+IGgCD8QFOEHgmKoD4X57LPPkvXLLktfhjIyMpKs33HHHWVrr7zySnLdSxlDfQCSCD8QFOEHgiL8QFCEHwiK8ANBEX4gqIrf50dsN9xwQ7J+//33J+s33nhj2VqlcfxKDh06lKzv37+/rtef6tjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPNPcUuWLEnWe3t7k/V77703WZ8zZ86ke6rW+fPnk/VSqZSsj46O5tnOlMOeHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqjjOb2bzJW2TNEfSqKR+d/+hmbVJ2iFpoaSjkh5w9w8b12pclcbSu7u7y9YqjeMvXLiwlpZyMTg4mKyvX78+Wd+1a1ee7YRTzZ5/RNK/uftXJX1d0hozu07So5JecvdrJL2UPQZwiagYfncvufuB7P5pSYclzZO0UtLWbLGtku5pVJMA8jepc34zWyjpa5IGJM1295I09gdC0tV5Nwegcaq+tt/MvijpOUnfc/dTZlVNByYz65HUU1t7ABqlqj2/mX1BY8H/mbs/nz193MzmZvW5kk5MtK6797t7h7t35NEwgHxUDL+N7eJ/Iumwu28aV9olaVV2f5Wknfm3B6BRKk7RbWbLJf1G0tsaG+qTpHUaO+//paQFkv4s6VvufrLCa4Wconv27NnJ+nXXXZesP/3008n6tddeO+me8jIwMJCsP/nkk2VrO3em9xd8Jbc21U7RXfGc391/K6nci90+maYAtA6u8AOCIvxAUIQfCIrwA0ERfiAowg8ExU93V6mtra1sra+vL7lue3t7sr5o0aKaesrDa6+9lqxv3LgxWd+zZ0+y/sknn0y6JzQHe34gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCCrMOP9NN92UrK9duzZZX7ZsWdnavHnzauopLx9//HHZ2ubNm5PrPvHEE8n62bNna+oJrY89PxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EFWacv6urq656PQ4dOpSs7969O1kfGRlJ1lPfuR8eHk6ui7jY8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUObu6QXM5kvaJmmOpFFJ/e7+QzN7TNJqSX/NFl3n7r+q8FrpjQGom7tbNctVE/65kua6+wEzmynpDUn3SHpA0hl3f6rapgg/0HjVhr/iFX7uXpJUyu6fNrPDkor96RoAdZvUOb+ZLZT0NUkD2VO9ZvZ7M9tiZleVWafHzAbNbLCuTgHkquJh/98WNPuipFclrXf3581stqQPJLmkH2js1OChCq/BYT/QYLmd80uSmX1B0m5Je9x90wT1hZJ2u/vSCq9D+IEGqzb8FQ/7zcwk/UTS4fHBzz4IvKBL0juTbRJAcar5tH+5pN9IeltjQ32StE5St6R2jR32H5X03ezDwdRrsecHGizXw/68EH6g8XI77AcwNRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCavYU3R9I+r9xj7+UPdeKWrW3Vu1Lorda5dnbP1a7YFO/z/+5jZsNuntHYQ0ktGpvrdqXRG+1Kqo3DvuBoAg/EFTR4e8vePsprdpbq/Yl0VutCumt0HN+AMUpes8PoCCFhN/M7jSzI2b2rpk9WkQP5ZjZUTN728wOFj3FWDYN2gkze2fcc21m9msz+1N2O+E0aQX19piZvZ+9dwfN7F8L6m2+mb1sZofN7A9m9kj2fKHvXaKvQt63ph/2m9k0SX+U1ClpSNLrkrrd/VBTGynDzI5K6nD3wseEzexfJJ2RtO3CbEhm9h+STrr7huwP51Xu/u8t0ttjmuTMzQ3qrdzM0t9Rge9dnjNe56GIPf8ySe+6+3vufk7SLyStLKCPlufu+yWdvOjplZK2Zve3auw/T9OV6a0luHvJ3Q9k909LujCzdKHvXaKvQhQR/nmS/jLu8ZBaa8pvl7TXzN4ws56im5nA7AszI2W3Vxfcz8UqztzcTBfNLN0y710tM17nrYjwTzSbSCsNOXzD3f9Z0l2S1mSHt6jOjyQt1tg0biVJG4tsJptZ+jlJ33P3U0X2Mt4EfRXyvhUR/iFJ88c9/rKkYwX0MSF3P5bdnpD0gsZOU1rJ8QuTpGa3Jwru52/c/bi7n3f3UUk/VoHvXTaz9HOSfubuz2dPF/7eTdRXUe9bEeF/XdI1ZvYVM5su6duSdhXQx+eY2YzsgxiZ2QxJ31TrzT68S9Kq7P4qSTsL7OXvtMrMzeVmllbB712rzXhdyEU+2VDGf0maJmmLu69vehMTMLNFGtvbS2PfePx5kb2Z2XZJt2rsW1/HJX1f0n9L+qWkBZL+LOlb7t70D97K9HarJjlzc4N6Kzez9IAKfO/ynPE6l364wg+IiSv8gKAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8E9f/Ex0YKZYOZcwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "img = feature_train[0]\n",
    "plt.imshow(img, cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   3  18  18  18 126 136\n",
      "  175  26 166 255 247 127   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  30  36  94 154 170 253 253 253 253 253\n",
      "  225 172 253 242 195  64   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0  49 238 253 253 253 253 253 253 253 253 251\n",
      "   93  82  82  56  39   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0  18 219 253 253 253 253 253 198 182 247 241\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  80 156 107 253 253 205  11   0  43 154\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0  14   1 154 253  90   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0 139 253 190   2   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0  11 190 253  70   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0  35 241 225 160 108   1\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0  81 240 253 253 119\n",
      "   25   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0  45 186 253 253\n",
      "  150  27   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  16  93 252\n",
      "  253 187   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 249\n",
      "  253 249  64   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0  46 130 183 253\n",
      "  253 207   2   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0  39 148 229 253 253 253\n",
      "  250 182   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0  24 114 221 253 253 253 253 201\n",
      "   78   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0  23  66 213 253 253 253 253 198  81   2\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0  18 171 219 253 253 253 253 195  80   9   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0  55 172 226 253 253 253 253 244 133  11   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0 136 253 253 253 212 135 132  16   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]\n",
      " [  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0   0   0   0   0]]\n"
     ]
    }
   ],
   "source": [
    "print(feature_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADQNJREFUeJzt3W+MVfWdx/HPZylNjPQBWLHEgnQb3bgaAzoaE3AzamxYbYKN1NQHGzbZMH2AZps0ZA1PypMmjemfrU9IpikpJtSWhFbRGBeDGylRGwejBYpQICzMgkAzJgUT0yDfPphDO8W5v3u5/84dv+9XQube8z1/vrnhM+ecOefcnyNCAPL5h7obAFAPwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+IKnP9HNjtrmdEOixiHAr83W057e9wvZB24dtP9nJugD0l9u9t9/2LEmHJD0gaVzSW5Iei4jfF5Zhzw/0WD/2/HdJOhwRRyPiz5J+IWllB+sD0EedhP96SSemvB+vpv0d2yO2x2yPdbAtAF3WyR/8pju0+MRhfUSMShqVOOwHBkkne/5xSQunvP+ipJOdtQOgXzoJ/1uSbrT9JduflfQNSdu70xaAXmv7sD8iLth+XNL/SJolaVNE7O9aZwB6qu1LfW1tjHN+oOf6cpMPgJmL8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaTaHqJbkmwfk3RO0seSLkTEUDeaAtB7HYW/cm9E/LEL6wHQRxz2A0l1Gv6QtMP2Htsj3WgIQH90eti/LCJO2p4v6RXb70XErqkzVL8U+MUADBhHRHdWZG+QdD4ivl+YpzsbA9BQRLiV+do+7Ld9te3PXXot6SuS9rW7PgD91clh/3WSfm370np+HhEvd6UrAD3XtcP+ljbGYT/Qcz0/7AcwsxF+ICnCDyRF+IGkCD+QFOEHkurGU30prFq1qmFtzZo1xWVPnjxZrH/00UfF+pYtW4r1999/v2Ht8OHDxWWRF3t+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iKR3pbdPTo0Ya1xYsX96+RaZw7d65hbf/+/X3sZLCMj483rD311FPFZcfGxrrdTt/wSC+AIsIPJEX4gaQIP5AU4QeSIvxAUoQfSIrn+VtUemb/tttuKy574MCBYv3mm28u1m+//fZifXh4uGHt7rvvLi574sSJYn3hwoXFeicuXLhQrJ89e7ZYX7BgQdvbPn78eLE+k6/zt4o9P5AU4QeSIvxAUoQfSIrwA0kRfiApwg8k1fR5ftubJH1V0pmIuLWaNk/SLyUtlnRM0qMR8UHTjc3g5/kH2dy5cxvWlixZUlx2z549xfqdd97ZVk+taDZewaFDh4r1ZvdPzJs3r2Ft7dq1xWU3btxYrA+ybj7P/zNJKy6b9qSknRFxo6Sd1XsAM0jT8EfELkkTl01eKWlz9XqzpIe73BeAHmv3nP+6iDglSdXP+d1rCUA/9PzeftsjkkZ6vR0AV6bdPf9p2wskqfp5ptGMETEaEUMRMdTmtgD0QLvh3y5pdfV6taTnu9MOgH5pGn7bz0p6Q9I/2R63/R+SvifpAdt/kPRA9R7ADML39mNgPfLII8X61q1bi/V9+/Y1rN17773FZScmLr/ANXPwvf0Aigg/kBThB5Ii/EBShB9IivADSXGpD7WZP7/8SMjevXs7Wn7VqlUNa9u2bSsuO5NxqQ9AEeEHkiL8QFKEH0iK8ANJEX4gKcIPJMUQ3ahNs6/Pvvbaa4v1Dz4of1v8wYMHr7inTNjzA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBSPM+Pnlq2bFnD2quvvlpcdvbs2cX68PBwsb5r165i/dOK5/kBFBF+ICnCDyRF+IGkCD+QFOEHkiL8QFJNn+e3vUnSVyWdiYhbq2kbJK2RdLaabX1EvNSrJjFzPfjggw1rza7j79y5s1h/44032uoJk1rZ8/9M0opppv8oIpZU/wg+MMM0DX9E7JI00YdeAPRRJ+f8j9v+ne1Ntud2rSMAfdFu+DdK+rKkJZJOSfpBoxltj9gesz3W5rYA9EBb4Y+I0xHxcURclPQTSXcV5h2NiKGIGGq3SQDd11b4bS+Y8vZrkvZ1px0A/dLKpb5nJQ1L+rztcUnfkTRse4mkkHRM0jd72COAHuB5fnTkqquuKtZ3797dsHbLLbcUl73vvvuK9ddff71Yz4rn+QEUEX4gKcIPJEX4gaQIP5AU4QeSYohudGTdunXF+tKlSxvWXn755eKyXMrrLfb8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AUj/Si6KGHHirWn3vuuWL9ww8/bFhbsWK6L4X+mzfffLNYx/R4pBdAEeEHkiL8QFKEH0iK8ANJEX4gKcIPJMXz/Mldc801xfrTTz9drM+aNatYf+mlxgM4cx2/Xuz5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiCpps/z214o6RlJX5B0UdJoRPzY9jxJv5S0WNIxSY9GxAdN1sXz/H3W7Dp8s2vtd9xxR7F+5MiRYr30zH6zZdGebj7Pf0HStyPiZkl3S1pr+58lPSlpZ0TcKGln9R7ADNE0/BFxKiLerl6fk3RA0vWSVkraXM22WdLDvWoSQPdd0Tm/7cWSlkr6raTrIuKUNPkLQtL8bjcHoHdavrff9hxJ2yR9KyL+ZLd0WiHbI5JG2msPQK+0tOe3PVuTwd8SEb+qJp+2vaCqL5B0ZrplI2I0IoYiYqgbDQPojqbh9+Qu/qeSDkTED6eUtktaXb1eLen57rcHoFdaudS3XNJvJO3V5KU+SVqvyfP+rZIWSTou6esRMdFkXVzq67ObbrqpWH/vvfc6Wv/KlSuL9RdeeKGj9ePKtXqpr+k5f0TsltRoZfdfSVMABgd3+AFJEX4gKcIPJEX4gaQIP5AU4QeS4qu7PwVuuOGGhrUdO3Z0tO5169YV6y+++GJH60d92PMDSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFJc5/8UGBlp/C1pixYt6mjdr732WrHe7PsgMLjY8wNJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUlznnwGWL19erD/xxBN96gSfJuz5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiCpptf5bS+U9IykL0i6KGk0In5se4OkNZLOVrOuj4iXetVoZvfcc0+xPmfOnLbXfeTIkWL9/Pnzba8bg62Vm3wuSPp2RLxt+3OS9th+par9KCK+37v2APRK0/BHxClJp6rX52wfkHR9rxsD0FtXdM5ve7GkpZJ+W0163PbvbG+yPbfBMiO2x2yPddQpgK5qOfy250jaJulbEfEnSRslfVnSEk0eGfxguuUiYjQihiJiqAv9AuiSlsJve7Ymg78lIn4lSRFxOiI+joiLkn4i6a7etQmg25qG37Yl/VTSgYj44ZTpC6bM9jVJ+7rfHoBeaeWv/csk/Zukvbbfqaatl/SY7SWSQtIxSd/sSYfoyLvvvlus33///cX6xMREN9vBAGnlr/27JXmaEtf0gRmMO/yApAg/kBThB5Ii/EBShB9IivADSbmfQyzbZjxnoMciYrpL85/Anh9IivADSRF+ICnCDyRF+IGkCD+QFOEHkur3EN1/lPR/U95/vpo2iAa1t0HtS6K3dnWztxtanbGvN/l8YuP22KB+t9+g9jaofUn01q66euOwH0iK8ANJ1R3+0Zq3XzKovQ1qXxK9tauW3mo95wdQn7r3/ABqUkv4ba+wfdD2YdtP1tFDI7aP2d5r+526hxirhkE7Y3vflGnzbL9i+w/Vz2mHSauptw22/7/67N6x/WBNvS20/b+2D9jeb/s/q+m1fnaFvmr53Pp+2G97lqRDkh6QNC7pLUmPRcTv+9pIA7aPSRqKiNqvCdv+F0nnJT0TEbdW056SNBER36t+cc6NiP8akN42SDpf98jN1YAyC6aOLC3pYUn/rho/u0Jfj6qGz62OPf9dkg5HxNGI+LOkX0haWUMfAy8idkm6fNSMlZI2V683a/I/T9816G0gRMSpiHi7en1O0qWRpWv97Ap91aKO8F8v6cSU9+MarCG/Q9IO23tsj9TdzDSuq4ZNvzR8+vya+7lc05Gb++mykaUH5rNrZ8Trbqsj/NN9xdAgXXJYFhG3S/pXSWurw1u0pqWRm/tlmpGlB0K7I153Wx3hH5e0cMr7L0o6WUMf04qIk9XPM5J+rcEbffj0pUFSq59nau7nrwZp5ObpRpbWAHx2gzTidR3hf0vSjba/ZPuzkr4haXsNfXyC7aurP8TI9tWSvqLBG314u6TV1evVkp6vsZe/MygjNzcaWVo1f3aDNuJ1LTf5VJcy/lvSLEmbIuK7fW9iGrb/UZN7e2nyicef19mb7WclDWvyqa/Tkr4j6TlJWyUtknRc0tcjou9/eGvQ27AmD13/OnLzpXPsPve2XNJvJO2VdLGavF6T59e1fXaFvh5TDZ8bd/gBSXGHH5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpP4CIJjqosJxHysAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "img_test = feature_test[0]\n",
    "plt.imshow(img_test, cmap='gray')\n",
    "print(label_test[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "print(feature_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "784\n"
     ]
    }
   ],
   "source": [
    "# Reshape the data - MLPs do not understand such things as '2D'.\n",
    "# Reshape to 28 x 28 pixels = 784 features\n",
    "img_vector_len = feature_train[0].shape[0]*feature_train[0].shape[0]\n",
    "print(img_vector_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "flat_train = feature_train.reshape(feature_train.shape[0],img_vector_len)\n",
    "flat_test = feature_test.reshape(feature_test.shape[0],img_vector_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784) (10000, 784)\n"
     ]
    }
   ],
   "source": [
    "print(flat_train.shape,flat_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   3  18  18  18 126 136 175  26 166 255\n",
      " 247 127   0   0   0   0   0   0   0   0   0   0   0   0  30  36  94 154\n",
      " 170 253 253 253 253 253 225 172 253 242 195  64   0   0   0   0   0   0\n",
      "   0   0   0   0   0  49 238 253 253 253 253 253 253 253 253 251  93  82\n",
      "  82  56  39   0   0   0   0   0   0   0   0   0   0   0   0  18 219 253\n",
      " 253 253 253 253 198 182 247 241   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0  80 156 107 253 253 205  11   0  43 154\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0  14   1 154 253  90   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0 139 253 190   2   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0  11 190 253  70   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  35 241\n",
      " 225 160 108   1   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0  81 240 253 253 119  25   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0  45 186 253 253 150  27   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0  16  93 252 253 187\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0 249 253 249  64   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0  46 130 183 253\n",
      " 253 207   2   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0  39 148 229 253 253 253 250 182   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0  24 114 221 253 253 253\n",
      " 253 201  78   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0  23  66 213 253 253 253 253 198  81   2   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0  18 171 219 253 253 253 253 195\n",
      "  80   9   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  55 172 226 253 253 253 253 244 133  11   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0 136 253 253 253 212 135 132  16\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0]\n"
     ]
    }
   ],
   "source": [
    "print(flat_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#normalize the data\n",
    "X_train = flat_train.astype('float32')/255\n",
    "X_test = flat_test.astype('float32')/255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.01176471 0.07058824 0.07058824 0.07058824\n",
      " 0.49411765 0.53333336 0.6862745  0.10196079 0.6509804  1.\n",
      " 0.96862745 0.49803922 0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.11764706 0.14117648 0.36862746 0.6039216\n",
      " 0.6666667  0.99215686 0.99215686 0.99215686 0.99215686 0.99215686\n",
      " 0.88235295 0.6745098  0.99215686 0.9490196  0.7647059  0.2509804\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.19215687\n",
      " 0.93333334 0.99215686 0.99215686 0.99215686 0.99215686 0.99215686\n",
      " 0.99215686 0.99215686 0.99215686 0.9843137  0.3647059  0.32156864\n",
      " 0.32156864 0.21960784 0.15294118 0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.07058824 0.85882354 0.99215686\n",
      " 0.99215686 0.99215686 0.99215686 0.99215686 0.7764706  0.7137255\n",
      " 0.96862745 0.94509804 0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.3137255  0.6117647  0.41960785 0.99215686\n",
      " 0.99215686 0.8039216  0.04313726 0.         0.16862746 0.6039216\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.05490196 0.00392157 0.6039216  0.99215686 0.3529412\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.54509807 0.99215686 0.74509805 0.00784314 0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.04313726\n",
      " 0.74509805 0.99215686 0.27450982 0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.13725491 0.94509804\n",
      " 0.88235295 0.627451   0.42352942 0.00392157 0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.31764707 0.9411765  0.99215686\n",
      " 0.99215686 0.46666667 0.09803922 0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.1764706  0.7294118  0.99215686 0.99215686\n",
      " 0.5882353  0.10588235 0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.0627451  0.3647059  0.9882353  0.99215686 0.73333335\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.9764706  0.99215686 0.9764706  0.2509804  0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.18039216 0.50980395 0.7176471  0.99215686\n",
      " 0.99215686 0.8117647  0.00784314 0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.15294118 0.5803922\n",
      " 0.8980392  0.99215686 0.99215686 0.99215686 0.98039216 0.7137255\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.09411765 0.44705883 0.8666667  0.99215686 0.99215686 0.99215686\n",
      " 0.99215686 0.7882353  0.30588236 0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.09019608 0.25882354 0.8352941  0.99215686\n",
      " 0.99215686 0.99215686 0.99215686 0.7764706  0.31764707 0.00784314\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.07058824 0.67058825\n",
      " 0.85882354 0.99215686 0.99215686 0.99215686 0.99215686 0.7647059\n",
      " 0.3137255  0.03529412 0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.21568628 0.6745098  0.8862745  0.99215686 0.99215686 0.99215686\n",
      " 0.99215686 0.95686275 0.52156866 0.04313726 0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.53333336 0.99215686\n",
      " 0.99215686 0.99215686 0.83137256 0.5294118  0.5176471  0.0627451\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.        ]\n"
     ]
    }
   ],
   "source": [
    "print(X_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 10) (60000,)\n"
     ]
    }
   ],
   "source": [
    "# Convert target classes to categorical ones\n",
    "from keras.utils import to_categorical\n",
    "y_train = to_categorical(label_train)\n",
    "y_test = to_categorical(label_test)\n",
    "print(y_train.shape,label_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "print(label_train[0],y_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 典型的逻辑回归概率分布曲线\n",
    "\n",
    "![mlp](images/mlp.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#set configurable parameters number of parameters(input_size),number of lables(num_labels),batch size,number of hidden units, dropout\n",
    "input_size = X_train.shape[1]\n",
    "num_labels = y_train.shape[1]\n",
    "batch_size = 128\n",
    "hidden_units = 256\n",
    "dropout = 0.45"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 256)               200960    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 256)               65792     \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                2570      \n",
      "=================================================================\n",
      "Total params: 269,322\n",
      "Trainable params: 269,322\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "#set up the model\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense,Activation,Dropout\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(hidden_units,input_dim=input_size,activation='relu'))\n",
    "model.add(Dropout(dropout))\n",
    "model.add(Dense(hidden_units,activation='relu'))\n",
    "model.add(Dropout(dropout))\n",
    "model.add(Dense(num_labels,activation='softmax'))\n",
    "model.summary()     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compile the model\n",
    "model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784) (60000, 10)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape,y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 6s 100us/step - loss: 0.4284 - acc: 0.8688\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 5s 77us/step - loss: 0.1924 - acc: 0.9422\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 5s 81us/step - loss: 0.1543 - acc: 0.9530\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 6s 99us/step - loss: 0.1287 - acc: 0.9608\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 6s 108us/step - loss: 0.1157 - acc: 0.9640\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 5s 91us/step - loss: 0.0996 - acc: 0.9683\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 7s 109us/step - loss: 0.0959 - acc: 0.9710\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 7s 115us/step - loss: 0.0873 - acc: 0.9726\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 7s 115us/step - loss: 0.0802 - acc: 0.9747\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 7s 112us/step - loss: 0.0765 - acc: 0.9761\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 7s 109us/step - loss: 0.0746 - acc: 0.9766\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 7s 124us/step - loss: 0.0701 - acc: 0.9779\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 7s 121us/step - loss: 0.0671 - acc: 0.9786\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 8s 130us/step - loss: 0.0635 - acc: 0.9799\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 7s 110us/step - loss: 0.0636 - acc: 0.9794\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 7s 118us/step - loss: 0.0599 - acc: 0.9806\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 7s 120us/step - loss: 0.0593 - acc: 0.9816\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 8s 127us/step - loss: 0.0574 - acc: 0.9812\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 8s 131us/step - loss: 0.0554 - acc: 0.9819\n",
      "Epoch 20/20\n",
      "60000/60000 [==============================] - 7s 124us/step - loss: 0.0547 - acc: 0.9823\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2600027cf60>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#train the model\n",
    "model.fit(X_train,y_train,epochs=20,batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 44us/step\n",
      "[0.07089545356004964, 0.9817]\n"
     ]
    }
   ],
   "source": [
    "#evaluate the model based on the test data\n",
    "score = model.evaluate(X_test,y_test,batch_size=batch_size)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 784)\n"
     ]
    }
   ],
   "source": [
    "sample = X_train[0]\n",
    "sample = sample.reshape(1,sample.shape[0])\n",
    "print(sample.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x2600090d8d0>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADgpJREFUeJzt3X+MVfWZx/HPs1j+kKI4aQRCYSnEYJW4082IjSWrxkzVDQZHrekkJjQapn8wiU02ZA3/VNNgyCrslmiamaZYSFpKE3VB0iw0otLGZuKIWC0srTFsO3IDNTjywx9kmGf/mEMzxbnfe+fec++5zPN+JeT+eM6558kNnznn3O+592vuLgDx/EPRDQAoBuEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxDUZc3cmJlxOSHQYO5u1SxX157fzO40syNm9q6ZPVrPawFoLqv12n4zmybpj5I6JQ1Jel1St7sfSqzDnh9osGbs+ZdJetfd33P3c5J+IWllHa8HoInqCf88SX8Z93goe+7vmFmPmQ2a2WAd2wKQs3o+8Jvo0OJzh/Xu3i+pX+KwH2gl9ez5hyTNH/f4y5KO1dcOgGapJ/yvS7rGzL5iZtMlfVvSrnzaAtBoNR/2u/uImfVK2iNpmqQt7v6H3DoD0FA1D/XVtDHO+YGGa8pFPgAuXYQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVfMU3ZJkZkclnZZ0XtKIu3fk0RTyM23atGT9yiuvbOj2e3t7y9Yuv/zy5LpLlixJ1tesWZOsP/XUU2Vr3d3dyXU//fTTZH3Dhg3J+uOPP56st4K6wp+5zd0/yOF1ADQRh/1AUPWG3yXtNbM3zKwnj4YANEe9h/3fcPdjZna1pF+b2f+6+/7xC2R/FPjDALSYuvb87n4suz0h6QVJyyZYpt/dO/gwEGgtNYffzGaY2cwL9yV9U9I7eTUGoLHqOeyfLekFM7vwOj939//JpSsADVdz+N39PUn/lGMvU9aCBQuS9enTpyfrN998c7K+fPnysrVZs2Yl173vvvuS9SINDQ0l65s3b07Wu7q6ytZOnz6dXPett95K1l999dVk/VLAUB8QFOEHgiL8QFCEHwiK8ANBEX4gKHP35m3MrHkba6L29vZkfd++fcl6o79W26pGR0eT9YceeihZP3PmTM3bLpVKyfqHH36YrB85cqTmbTeau1s1y7HnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOfPQVtbW7I+MDCQrC9atCjPdnJVqffh4eFk/bbbbitbO3fuXHLdqNc/1ItxfgBJhB8IivADQRF+ICjCDwRF+IGgCD8QVB6z9IZ38uTJZH3t2rXJ+ooVK5L1N998M1mv9BPWKQcPHkzWOzs7k/WzZ88m69dff33Z2iOPPJJcF43Fnh8IivADQRF+ICjCDwRF+IGgCD8QFOEHgqr4fX4z2yJphaQT7r40e65N0g5JCyUdlfSAu6d/6FxT9/v89briiiuS9UrTSff19ZWtPfzww8l1H3zwwWR9+/btyTpaT57f5/+ppDsveu5RSS+5+zWSXsoeA7iEVAy/u++XdPElbCslbc3ub5V0T859AWiwWs/5Z7t7SZKy26vzawlAMzT82n4z65HU0+jtAJicWvf8x81sriRltyfKLeju/e7e4e4dNW4LQAPUGv5dklZl91dJ2plPOwCapWL4zWy7pN9JWmJmQ2b2sKQNkjrN7E+SOrPHAC4hFc/53b27TOn2nHsJ69SpU3Wt/9FHH9W87urVq5P1HTt2JOujo6M1bxvF4go/ICjCDwRF+IGgCD8QFOEHgiL8QFBM0T0FzJgxo2ztxRdfTK57yy23JOt33XVXsr53795kHc3HFN0Akgg/EBThB4Ii/EBQhB8IivADQRF+ICjG+ae4xYsXJ+sHDhxI1oeHh5P1l19+OVkfHBwsW3vmmWeS6zbz/+ZUwjg/gCTCDwRF+IGgCD8QFOEHgiL8QFCEHwiKcf7gurq6kvVnn302WZ85c2bN2163bl2yvm3btmS9VCrVvO2pjHF+AEmEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxXF+M9siaYWkE+6+NHvuMUmrJf01W2ydu/+q4sYY57/kLF26NFnftGlTsn777bXP5N7X15esr1+/Pll///33a972pSzPcf6fSrpzguf/093bs38Vgw+gtVQMv7vvl3SyCb0AaKJ6zvl7zez3ZrbFzK7KrSMATVFr+H8kabGkdkklSRvLLWhmPWY2aGblf8wNQNPVFH53P+7u5919VNKPJS1LLNvv7h3u3lFrkwDyV1P4zWzuuIddkt7Jpx0AzXJZpQXMbLukWyV9ycyGJH1f0q1m1i7JJR2V9N0G9gigAfg+P+oya9asZP3uu+8uW6v0WwFm6eHqffv2JeudnZ3J+lTF9/kBJBF+ICjCDwRF+IGgCD8QFOEHgmKoD4X57LPPkvXLLktfhjIyMpKs33HHHWVrr7zySnLdSxlDfQCSCD8QFOEHgiL8QFCEHwiK8ANBEX4gqIrf50dsN9xwQ7J+//33J+s33nhj2VqlcfxKDh06lKzv37+/rtef6tjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPNPcUuWLEnWe3t7k/V77703WZ8zZ86ke6rW+fPnk/VSqZSsj46O5tnOlMOeHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqjjOb2bzJW2TNEfSqKR+d/+hmbVJ2iFpoaSjkh5w9w8b12pclcbSu7u7y9YqjeMvXLiwlpZyMTg4mKyvX78+Wd+1a1ee7YRTzZ5/RNK/uftXJX1d0hozu07So5JecvdrJL2UPQZwiagYfncvufuB7P5pSYclzZO0UtLWbLGtku5pVJMA8jepc34zWyjpa5IGJM1295I09gdC0tV5Nwegcaq+tt/MvijpOUnfc/dTZlVNByYz65HUU1t7ABqlqj2/mX1BY8H/mbs/nz193MzmZvW5kk5MtK6797t7h7t35NEwgHxUDL+N7eJ/Iumwu28aV9olaVV2f5Wknfm3B6BRKk7RbWbLJf1G0tsaG+qTpHUaO+//paQFkv4s6VvufrLCa4Wconv27NnJ+nXXXZesP/3008n6tddeO+me8jIwMJCsP/nkk2VrO3em9xd8Jbc21U7RXfGc391/K6nci90+maYAtA6u8AOCIvxAUIQfCIrwA0ERfiAowg8ExU93V6mtra1sra+vL7lue3t7sr5o0aKaesrDa6+9lqxv3LgxWd+zZ0+y/sknn0y6JzQHe34gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCCrMOP9NN92UrK9duzZZX7ZsWdnavHnzauopLx9//HHZ2ubNm5PrPvHEE8n62bNna+oJrY89PxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EFWacv6urq656PQ4dOpSs7969O1kfGRlJ1lPfuR8eHk6ui7jY8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUObu6QXM5kvaJmmOpFFJ/e7+QzN7TNJqSX/NFl3n7r+q8FrpjQGom7tbNctVE/65kua6+wEzmynpDUn3SHpA0hl3f6rapgg/0HjVhr/iFX7uXpJUyu6fNrPDkor96RoAdZvUOb+ZLZT0NUkD2VO9ZvZ7M9tiZleVWafHzAbNbLCuTgHkquJh/98WNPuipFclrXf3581stqQPJLmkH2js1OChCq/BYT/QYLmd80uSmX1B0m5Je9x90wT1hZJ2u/vSCq9D+IEGqzb8FQ/7zcwk/UTS4fHBzz4IvKBL0juTbRJAcar5tH+5pN9IeltjQ32StE5St6R2jR32H5X03ezDwdRrsecHGizXw/68EH6g8XI77AcwNRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCavYU3R9I+r9xj7+UPdeKWrW3Vu1Lorda5dnbP1a7YFO/z/+5jZsNuntHYQ0ktGpvrdqXRG+1Kqo3DvuBoAg/EFTR4e8vePsprdpbq/Yl0VutCumt0HN+AMUpes8PoCCFhN/M7jSzI2b2rpk9WkQP5ZjZUTN728wOFj3FWDYN2gkze2fcc21m9msz+1N2O+E0aQX19piZvZ+9dwfN7F8L6m2+mb1sZofN7A9m9kj2fKHvXaKvQt63ph/2m9k0SX+U1ClpSNLrkrrd/VBTGynDzI5K6nD3wseEzexfJJ2RtO3CbEhm9h+STrr7huwP51Xu/u8t0ttjmuTMzQ3qrdzM0t9Rge9dnjNe56GIPf8ySe+6+3vufk7SLyStLKCPlufu+yWdvOjplZK2Zve3auw/T9OV6a0luHvJ3Q9k909LujCzdKHvXaKvQhQR/nmS/jLu8ZBaa8pvl7TXzN4ws56im5nA7AszI2W3Vxfcz8UqztzcTBfNLN0y710tM17nrYjwTzSbSCsNOXzD3f9Z0l2S1mSHt6jOjyQt1tg0biVJG4tsJptZ+jlJ33P3U0X2Mt4EfRXyvhUR/iFJ88c9/rKkYwX0MSF3P5bdnpD0gsZOU1rJ8QuTpGa3Jwru52/c/bi7n3f3UUk/VoHvXTaz9HOSfubuz2dPF/7eTdRXUe9bEeF/XdI1ZvYVM5su6duSdhXQx+eY2YzsgxiZ2QxJ31TrzT68S9Kq7P4qSTsL7OXvtMrMzeVmllbB712rzXhdyEU+2VDGf0maJmmLu69vehMTMLNFGtvbS2PfePx5kb2Z2XZJt2rsW1/HJX1f0n9L+qWkBZL+LOlb7t70D97K9HarJjlzc4N6Kzez9IAKfO/ynPE6l364wg+IiSv8gKAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8E9f/Ex0YKZYOZcwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "img_sample = feature_train[0]\n",
    "plt.imshow(img_sample, cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2.0128218e-18 9.6623664e-14 4.1070319e-15 1.7165091e-04 3.1745351e-18\n",
      "  9.9982834e-01 2.0662960e-15 1.9398024e-15 2.2530526e-12 2.0640423e-08]]\n"
     ]
    }
   ],
   "source": [
    "sample_predict = model.predict(sample)\n",
    "print(sample_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the predicted label is: 5\n",
      "the real labels is: 5\n"
     ]
    }
   ],
   "source": [
    "print('the predicted label is:',sample_predict.argmax())\n",
    "print('the real labels is:',label_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predict = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_test_predict = test_predict.argmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 10)\n"
     ]
    }
   ],
   "source": [
    "print(test_predict.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAELCAYAAAB3QSUaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmUVMUZ9/HvIwIqKAICigsgxARFZRFFA6KRV0DBgHskRtEoJmqIKC64K8aoiR41gjER97gvcQ/Gg+CCRoiCoMQXeEE4gIIIsogI1PtHT82dHnpgevp23dszv885fei+fbu6euaZ4um6tZhzDhERKa5tkq6AiEhdoMZWRCQANbYiIgGosRURCUCNrYhIAGpsRUQCCN7Ymtm9ZnZ13OdKaVNcSC61KS4sznG2ZjYPaAVsADYCnwIPA/c55zYVWPYRwKPOuT3yeM1rQK8KhxoA/3PO7V9IXSQ/KYyLkcAZQBtgGTDGOXdbIfWQ/KUwLo4ErgG6At8459oWUofKipHZDnTO7UgmkP8IXAbcX4T32SrnXH/nXGN/A94Dnk6iLpKeuAAM+BXQFOgHXGBmpyZUl7ouTXGxBhgHjCxK6c652G7APKBPpWMHA5uATmWPHwRGV3j+UmAxsAj4NeCADhXPBRoB35WVs7rs1jrPurUl879nuzg/s26lHRdl5d0F3J30z6mu3dIaF0AfYF7cn7fofbbOuf8AC8n+Og+AmfUDRpR9uA5A7yrKWAP0Bxa5KFNdZGY9zWxFNavyK+Bt59z/q8nnkHilJS7MzMrqMLNmn0TilJa4KIZQF8gWAc1yHD8ZeMA5N9M5txa4Pp9CnXPvOOd2rubpvyLzP5+kRxri4joyfwcP5PMeUlRpiIvYhWpsdweW5zjeGlhQ4fGCHOcUzMx6ArsCzxSjfKmxpOPiAjL/CR/rnPu+GO8hNZJoXBTLtsV+AzPrTuaH906OpxcDFa8W7rmFogoZNnEG8JxzbnUBZUiMko4LMzsLuBw43Dm3sCZlSPySjotiKlpma2Y7mdkA4AkyQzA+yXHaU8BQM+toZjuQGXZRlS+B5mbWJM96bA+chLoQUiENcWFmQ4A/AP/HOTc3j+pLkaQkLrYxs+2A+pmHtp2ZNcjjY2xRMRrbl8xsFZkU/0rgdmBorhOdc6+RuRI8AZgNTC57arOvdM65WcDjwFwzW2Fmrc2sl5ltLVsdBKwsew9JTpriYjTQHPjQzFaX3e6t6QeTgqQpLg4nM4rhVWCvsvvja/Spcoh1UkOhzKwjMANo6JzbkHR9JB0UF5JLqcVF4msjmNlgM2tgZk2BW4CXSuEHJ8WluJBcSjkuEm9sgWHAUmAOmUkHv0m2OpISigvJpWTjIlXdCCIitVUaMlsRkVpPja2ISAB5TWowszrR5+Ccs6TrUErqSlwAy5xzLZKuRKlQXGRTZitSffOTroCkUrXiQo2tiEgAamxFRAJQYysiEoAaWxGRAIq+xKJITVxyySUAbL/99gAccMABAJx44olZ540dOxaAyZMza5I88sgjoaookhdltiIiAeQ1XbeujJvTONv8xBkXTz75JLB5Brs1c+bMAaBPnz4AfPHFF3FVqaKpzrmDilFwbZSG9mKfffYBYNasWQAMHz4cgLvvvjvOt6lWXCizFREJQH22kgpby2h9ZvKvf/0LgL333huAgQMHAtC+fXsAhgwZAsDNN99cvMpKyejSpQsAmzZtAmDhwuR2QFJmKyISgDJbScxBB0XdXIMHD856bubMmQAcd9xxACxbtgyA1aszu5o0aJDZGur9998H4MADDwSgefPmRayxlJrOnTsDsGbNGgCef/75xOqizFZEJICiZra+/+2cc84pP7Zo0SIA1q1bB8Bjjz0GwJIlSwCYPXt2MaskKbLbbruV3zfLDADxGW3fvn0BWLx4cc7XXnzxxQDsu+++WcdfeeWV2OsppadTp04AXHDBBUA6xl8rsxURCUCNrYhIAEXtRrj11lsBaNu2bZXnDBs2DIBVq1YB0dfImvJDO/x7A0yZMqWgMqU4XnrppfL7HTp0AKI4WL58+RZfe+qppwJQv379ItVOStlPfvITABo1agREQwuTpMxWRCSAoma2/sKYX0QE4LPPPgOgY8eOAHTt2hWAI444AoAePXoAsGDBAgD23HPPnGVv2JDZKn7p0qVA9sUWyJ6uqcw2/ebPr94mCCNHjgSiaZjeBx98kPWv1G2XXnopEMVVGtoAZbYiIgEUNbN98803s/6t6PXXX8963LRpUyAahDx16lQAunfvnrNsP3Ts888/B6KMuVmzZkC0MInUDgMGDADghhtuAKJJDV999RUAV1xxBQBr165NoHaSFv76kJ8w49sHP6khScpsRUQCSM103W+++QaACRMmZB3PlRVXdMIJJwBRZvzJJ58A6bj6KPHxmYrPaD3/e544cWLwOkn69O7dO+uxv6aTBspsRUQCSE1mm6+WLVsCMGbMGAC22Sbz/4bv09vaOE0pDS+88AIARx99dNbxhx9+GICrrroqeJ0kvfbff/+sxxXH2ydNma2ISAAlm9mef/75ALRo0QKI+nz/97//JVYniY8fN33YYYcB0LBhQyBaanH06NFAtOSi1G1+fP7QoUMB+OijjwB44403EqtTZcpsRUQCKLnM9qc//SkAl19+edbxQYMGATBjxozgdZL4Pfvss8Dmi4E/+uijgMZRSza/0acfZ+/H8fvx+GmgzFZEJICSy2yPOeYYIFrtyY/DnTx5cmJ1kvj4bXD8mhneW2+9BcC1114bukpSAvy2SM5ldk9/5plnkqxOTspsRUQCKJnMdvvttwegX79+AKxfvx6IMp0ffvghmYpJLHzf7KhRo4DN16n9+OOPAY0+kGy77rorAL169QKi0UhJbuxYFWW2IiIBlExm69cx7dKlCxBdbXzvvfcSq5PEx2/gWHmVNz+DTH21ksuZZ54JRDNKX3vttQRrs2XKbEVEAkh9ZnvssccCcPXVVwPw7bffAtEaCFI7jBgxIudxvxW1+mollzZt2mQ99jNJ00iZrYhIAKnNbP3V6bvuuguAevXqAfDqq68C8P777ydTMQnKzwja2miTlStXZp3nRzM0adIk67ydd965/H5V2fTGjRsBuOyyywDt/pBmfgcPr+KOzWmjzFZEJIDUZbY+g/WjDdq1awdEc+F9363UDdOnT6/WeU8//TQAixcvBqBVq1YAnHLKKTV+7yVLlgBw00031bgMKY6ePXsC0TjbUqDMVkQkgNRltu3btwegW7duWcd9/5pWe6qdfF/8z3/+8xq9/qSTTtri8xs2bABg06ZNmz334osvAjBlypSs42+//XaN6iLFN3jwYCD6JuzXr500aVJiddoaZbYiIgGosRURCSA13Qh+cPL48eOzjvtpui+//HLwOkk4xx9/PACXXnopsPlCNN5+++0HVH3ha9y4cQDMmzcv67hfjHzWrFkF11WSs8MOOwDRUqueX1LRD9tLI2W2IiIBpCazPffccwHYa6+9so5PnDgRiBYFltqtultPn3baaUWuiaSRn7Tip+X6i5t33nlnYnWqLmW2IiIBJJ7Z+sHJF154YcI1EZG085mt3+K+lCizFREJIPHM1m9n0bhx46zjfvKCltYTkdpAma2ISACJZ7aVTZs2DYCjjjoKgOXLlydZHRGRWCizFREJwPIZv2pmdWKwq3POkq5DKakrcQFMdc4dlHQlSoXiIpsyWxGRAPLts10GzC9GRVKkzdZPkUrqQlyAYiNfiosK8upGEBGRmlE3gohIAGpsRUQCUGMrIhKAGlsRkQDU2IqIBKDGVkQkADW2IiIBqLEVEQlAja2ISABqbEVEAlBjKyISgBpbEZEAgje2ZnavmV0d97lS2hQXkkttiotYV/0ys3lAK2ADsBH4FHgYuM85t6nAso8AHnXO7ZHHawz4I/DrskP3A5c5LXUWVNriosJrGwDTgcY1eb0UJm1xYWZHAtcAXYFvnHNtC6lDZcXIbAc653Yks8bjH4HLyDRySTgXGAQcCBwADACGJVSXui5NceGNBL5KuA51XZriYg0wjkxcxM85F9sNmAf0qXTsYGAT0Kns8YPA6ArPXwosBhaRyUAd0KHiuUAj4LuyclaX3VpXoz7vAedWeHw28H6cn1m30ouLsjLaAZ8B/YGFSf+M6uItjXFRVk4fYF7cn7fofbbOuf8AC4FelZ8zs37AiLIP1wHoXUUZa8j8USxyzjUuuy0ys55mtmILb78fMK3C42llxyRhCccFwN3AKDJ/lJISKYiLogl1gWwR0CzH8ZOBB5xzM51za4Hr8ynUOfeOc27nLZzSGFhZ4fFKoHFZX64kL5G4MLPBwLbOuefzqq2EklR7UVShGtvdgeU5jrcGFlR4vCDHOYVYDexU4fFOwGpX9l1BEhc8LsysEXArcGFcZUrskmoviqroja2ZdSfzw3snx9OLgYpXC/fcQlE1aSBnkrk45h1YdkwSlmBc/AhoC7xtZkuA54DdzGyJmbXNsyyJWcLtRVEVrbE1s53MbADwBJkhGJ/kOO0pYKiZdTSzHcgMu6jKl0BzM2uSRzUeBkaY2e5m1hq4mEwnuiQkBXExg8wfaeey26/LyuhMiWVKtUkK4gIz28bMtgPqZx7admXDA2NRjMb2JTNbRSZwrwRuB4bmOtE59xpwFzABmA1MLnvq+xznzgIeB+aa2Qoza21mvcxs9Rbq8lfgJeATMn9kr5Qdk/BSERfOuQ3OuSX+Rubr6qayxxsL/IySv1TERZnDyVwwfRXYq+z++Bp9qhxStZW5mXUk0yg2dM5tSLo+kg6KC8ml1OIi8bURzGywmTUws6bALcBLpfCDk+JSXEgupRwXiTe2ZGZ0LQXmkJmy95tkqyMpobiQXEo2LlLVjSAiUlulIbMVEan11NiKiASwbT4nm1md6HNwzmk6bx7qSlwAy5xzLZKuRKlQXGRTZitSffOTroCkUrXiQo2tiEgAamxFRAJQYysiEoAaWxGRANTYiogEkNfQrxAaNWoEwG233QbAsGGZ/RmnTp0KwEknnQTA/Pm6MCwipUOZrYhIAKnLbHfbbTcAzjnnHAA2bcpsH9+tWzcABgwYAMA999yTQO0klK5duwLw3HPPAdC2bdsalXP00UcD8NlnnwGwYIHWB69LBg4cCMCLL74IwAUXXADAvffeC8DGjeGWMFZmKyISQGoy2xYtMrPdHnrooYRrImnQt29fABo2bFhQOT6zOeusswA49dRTC6uYlITmzZsDMGbMmKzjf/nLXwAYN24cAN99F24ne2W2IiIBJJ7Z/u53vwNg0KBBABx88MFbPP/www8HYJttMv9PTJs2DYBJkyYVq4oS0LbbZkLymGOOiaU8P4plxIgRQDTaBWDNmjWxvIekj28n9thjj6zjjz/+OADr1q0LXidltiIiAaixFREJIPFuhDvuuAOIhnhtzfHHH5/1r5/ccMoppwDR10YpTUceeSQAhx56KAC33nprQeU1bdoUgH333ReAHXbYofw5dSPUPv6C6pVXXpnz+UceeQSAJLYDU2YrIhJAXhs+xrny+quvvgpA//79ga1ntl9//TUAq1evBqBNmzY5z6tXr17BddNODfmJIy46deoEwFtvvQVEv28/mcX/3vPly+vZsycQTZoBWLp0ab7FTXXOHVSjitRBSezUcNBBmV/Phx9+mHV8w4bMbuf169cvxttWKy6U2YqIBBC8z7Z3794A/PjHPwaijLaqzNZPqxs/fjwAK1euBOBnP/sZsHnfzG9+k9lGfuzYsXFWW4rsqquuAqKhWf369QNqntE2a9YMiOKtutcEpLSdcMIJOY/79iNJymxFRAIIktlWXETkiSeeAGCXXXbJea4fXfDss88CcP311wOwdu3anOede+65QDTd11+93m677crP9VP0fvjhh5p/CIndiSeeWH7fT2KYPXs2AFOmTCmobP+Nx2e0vu92xYoVBZUr6eYnM3jr168Hqh6dEJIyWxGRAIJktn4KJlSd0U6cOBGIFgpZtmzZFsv0me3NN98MwO233w5E4ygrjs/0y6vNmTMn77pL8fiF4CH6vVVeOCRf/lvUkCFDgGgJvdGjRwP6dlNbHXbYYVn/en4s9ccffxy8TpUpsxURCSDxGWS+b84vgbe1jLYyn7X6TKZ79+4x1k6KoUmTJgD06NFjs+cKHUXi+/D9Nyi/aPiECRMKKlfSraq/+zSNSlJmKyISQPDM1i+N6B1yyCEFlWdmWeVWLh/guuuuA+D0008v6L0kHn7++u67715+zC99V6j27dtnPZ4xY0Ys5Uq6+Zljnh91osxWRKSOCZLZnnfeeeX3457J47c96dKlS1b5Fd/HZ7aSDqtWrQKyrxAfcMABQDTza/ny5XmV2bJlSyB77C7AO++8U+N6Svr5NS9OO+20rON+punChQuD16kqymxFRAIIktn67DMOfqaYX5901KhROc+ruKKTxlami99kr+K4Zz+n/ZVXXgGicdNV8auE7b333kA0vrbyKnZaE6F28xs7Vr5W88YbbyRRnS1SZisiEkDi42zz5ec4n3/++TmfnzdvHgBnnHFG+bEvvvii6PWS/F177bXl9/2okmOPPRbY+ugEPx7bZ7JVzUx88MEHC62mpFjlPno/CuGvf/1rEtXZImW2IiIBlExm63d28OvgVuXTTz8FdBW6FMyaNav8/sknnwxA586dAejQocMWX/vMM89kPX7ooYeAaCah5/uHpXbxW5RXHoXgRx8UumpcMSizFREJIEhm6/vjYPOrhn4PMu++++4DoHXr1lnH/eu2dnU5zpEPEp4fe5vvKk1z587NedyPWtBMstrFr+5VuT154YUXkqhOtSizFREJIEhmW3F+csV1ZgFefvllYPOMtaoMdmt7lUnd5L89VfwWBcpoays/vtbzo1PuvPPOJKpTLcpsRUQCCJLZPvfcc+X3R44cCUQzwfLlZ4b5dUr9+qWLFy8upIpS4vx428ozyKR26tu3b9ZjP5ber4mQRspsRUQCUGMrIhJAkG4EvzkjRBs6Dho0CIDhw4fnVdZNN90EwD333BNT7aQ2qLh1PWgyQ21Vv359YPNF4tetWweke9EpZbYiIgEEn647adKkrH/Hjx8PRBe6/KQEv5Gjn+Tgh/T46bgiFQ0dOhSIFiK58cYbk6yOFIkf+umn4/pJK7Nnz06sTtWlzFZEJIDEF6J5/fXXs/4VqYkPP/wQiBYd19bltdPGjRuBaKlVP9Rv6tSpidWpupTZiogEYPkMAjezOjFi3DlnWz9LvLoSF8BU59xBWz9NQHFRmTJbEZEA1NiKiASgxlZEJAA1tiIiAaixFREJIN9xtsuA+Vs9q7S1SboCJaguxAUoNvKluKggr6FfIiJSM+pGEBEJQI2tiEgAamxFRAJQYysiEoAaWxGRANTYiogEoMZWRCQANbYiIgGosRURCUCNrYhIAGpsRUQCUGMrIhKAGlsRkQCCN7Zmdq+ZXR33uVLaFBeSS22Ki1iXWDSzeUArYAOwEfgUeBi4zzm3qcCyjwAedc7tUYPXNgCmA41r8nopTNriwsx2Bu4E+pcdGuOcu66Qekj+UhgX1wFXAt9XOHyAc25uIXXxipHZDnTO7UhmQd0/ApcB9xfhffIxEvgq4TrUdWmKizuAHYC2wMHA6WY2NKG61HVpiguAJ51zjSvcYmlooYjdCM65lc65F4FTgDPMrBOAmT1oZqP9eWZ2qZktNrNFZvZrM3Nm1qHiuWbWCHgNaG1mq8turatTDzNrB/wSuDnuzyj5S0lcDARudc6tdc7NI/PHfVbMH1XykJK4KKqi99k65/4DLAR6VX7OzPoBI4A+QAegdxVlrCHzlW9Rhf9xFplZTzNbsZUq3A2MAr4r4GNIzFIQF1bpfqcafAyJWQriYqCZLTezmWb2m4I+TCWhLpAtAprlOH4y8IBzbqZzbi1wfT6FOufecc7tXNXzZjYY2NY593xetZVQEokL4HXgcjPbsSwrOotMt4KkQ1Jx8RTQEWgBnANcY2a/yOc9tiRUY7s7sDzH8dbAggqPF+Q4p0bKvkrcClwYV5kSu+BxUeZ3ZL7p/F/gn8DjZLIpSYdE4sI596lzbpFzbqNz7j0yF1FPjKv8fHfXzZuZdSfzw3snx9OLgYpXC/fcQlH5Dpv4EZkLIG+bGUADoImZLQF6lPXVSUISjAucc8uBIRXq8gfgP/mWI/FLMi6qKMO2elY1FS2zNbOdzGwA8ASZIRif5DjtKWComXU0sx2Aa7ZQ5JdAczNrUs0qzCDzy+hcdvt1WRmdiT9TkmpKQVxgZu3NrLmZ1TOz/sC5wOitvU6KJyVx8XMza2oZB5P5BvTPPD7GFhWjsX3JzFaRadCuBG4Hcg6rcc69BtwFTABmA5PLnvo+x7mzyHzdm2tmK8ystZn1MrPVVZS9wTm3xN/IfC3ZVPZ4Y4GfUfKXirgo0w34BFhFZpTKEOfczJp9LClQmuLi1LJyV5EZ73uLc+6hmn2szcU6qaFQZtaRTEba0Dm3Ien6SDooLiSXUouLxNdGMLPBZtbAzJoCtwAvlcIPTopLcSG5lHJcJN7YAsOApcAcMlP2Yh3bJiVLcSG5lGxcpKobQUSktkpDZisiUuupsRURCSCvSQ1mVif6HJxzsQ1krgvqSlwAy5xzLZKuRKlQXGRTZitSffOTroCkUrXiQo2tiEgAamxFRAJQYysiEoAaWxGRANTYiogEUPT1bEVEkta0aVMA9tprr5zPz58fDSi46KKLAJgxYwYAn3/+OQDTpk0rqA7KbEVEAkgss23ZsiUATz31FADvvfceAPfddx8A8+bNK6j8Jk2iNYMPP/xwAF5//XUAfvjhh4LKFpF0O/bYYwE47rjjADjiiCMA6NChQ87zffYK0KZNGwAaNmyYdU69evUKqpMyWxGRAIJntr7vZObMzML4PgP98ssvgfgy2qlTp5Yfa9EiM5OuW7duAMyePbug95BwdtppJwBuvvlmADp1yuw43qdPH0DfUuq69u3bA3D++ecDcM455wCw/fbbA1C2/+BW7bPPPkWoXTZltiIiAQTJbHfZZZfy+08++SQAzZpltoUfM2YMABdeGM+O41dddRUA7dq1Kz82bNgwQBltKRkyJLP57U033QTAnntmb6TqM96vv/46bMUkVfbYI7PZ7vDhw2v0+lmzZgHRN+1iUmYrIhKAGlsRkQDy2hanputTHn300eX3X3vttazndt11VwCWLl1ak6LL7bfffgB88klmu/nnn3++/LkzzzwTgFWrVlWrLK1nm5841y31Xws/+ugjAJo3bw5A5Tj13VEXXHABAMuXL4+rClsy1Tl3UIg3qg3iiAvfBem7Cd59910gGsbZo0cPAF599VUA1qxZA0CjRo0AGD9+PBBNUPjggw+AKL6+++67rNfVULXiQpmtiEgARb1A5icunHDCCZs9d/bZZwPxZbT//ve/s45XzGyrm9FK8i655BIguoBalVNOOQWAfv36AdGFtLvvvhuA9evXF6uKUmQ+K4UoMz3wwAMBGDx4cNa577//PgBdu3YFoqGjflruwoULAdi0aVPxKlxNymxFRAIoamb75z//GYBf/vKX5cf8ZIOnn346lvfo1asXAK1atQLgwQcfBODRRx+NpXwJw0+RHDp0aNbx6dOnA9GkFz+ZwfOTWHxG/NhjjwGwZMmS4lVWiqJBgwYA/OMf/yg/5jPaP/zhD8Dm32C9ypOhvvjiiyLUsDDKbEVEAihqZuuvIFfsL1m0aBFQ8z41Pw1v1KhRAPz2t7/Neq+zzjqrZpWVRHXu3BmAHXfcEYC3334bgN69ewOw3XbbAfCLX/wCiH7/frqmH9Xyz3/+E4D+/fsDwUYpSAEaN24MwBVXXAHAgAEDyp9btmwZAH/6058AWLt2beDaxUeZrYhIAMEXovFLn/mrjCtWrABg7NixW3ydz3D8Uml+fJ33zDPPxFlNCcwvZ+e/odxxxx1Zz69btw6ABx54AICTTjoJgL333jvrPJ/5aDRC6Rg0aBAAl19+OZDd3+qvyaxcuTJ8xWKmzFZEJICiZrZ33nknAEceeWT5sdatWwPRgt5+CTS/yG9V/HmVZxLNnTsXiPrwpDT5vljPfwN64YUXcp5/0EG5J+z4cZerV6+OsXZSTIcddljWYz+7C6JxsrWBMlsRkQCCrI3gFwyH6Kqzn/kzcuRIAL766isAHnrooZxlPPLII8Dmm6758bRnnHFGTaqWk9ZGyE8cc+BPPvlkAB5//HEgWuPi1FNPBWD//fcHohlEvs/222+/BaIY86MP/DenTz/9tNCqVaS1EfJQ3bjwf/t+HYzvv/++/LlbbrkFiEaZfPzxx7HWMSZaG0FEJC2CZLZx8Fed/QLg/n+4vn37AoWvsVCRMtv8xBEXfi0E//v1M8Oq6qv3M4n8digvv/wyAD/60Y8A+Nvf/gbAeeedV2jVKlJmm4fqxkWu8fiV+efuvfdeIOqb92sg+LipvAi4Xztl8uTJQNH6gJXZioikRclktn7Ng9NPPx2I+nzfeOON2N9LmW1+4owLv/aBHzftM1wfp35Vr8suuwyIxt/6ufN+rOb8+fOzypszZ04c1VNmm4fqxsVtt90GwIgRI4pWF//N96233gKiawExUWYrIpIWqc9s/VVnvzK/X5vWj93973//G/t7KrPNTzHiwmekp512GhDNNLzmmmuAzcfR+jUz/IpRftx2zKNVlNnmobpxUa9ePQC6dOkCZK/6te22makAfsPPbbYpLD/07d11110HwOjRowsqr4wyWxGRtAi+NkK+/OpNnr/qXIyMVtLDjzaoav3SyvxeUv4bkM9s/TcgP9pBq4Clz8aNGwGYMmUKAPvss89m5xx11FEA1K9fH4gy0+7du+f1Xn50S7du3WpU10IosxURCaBkMlu/+6Xf/UEkl6eeegqIMlu/V5nfhfeGG25IpmJSkDfffDPrsZ+J6jPbDRs2ANGqcH6c9e9//3sg6vtPkjJbEZEAUjsawc/8GTNmDBDNn/Yr8heTRiPkJ8nx11Xxmc+7774LRDs9dOzYsfyczz//PN9iNRohD8WMC7+b7ocffpjz+QkTJgDR+tdfEUxRAAACA0lEQVS+r9bz7cqFF14YR3U0GkFEJC1Sm9n6tQ/8ak9+BtnZZ58NRHtV+dWe4txNU5ltftKY2XoXX3wxEM1Seu6558qf87MR/UiGalBmm4dixoUfVz1u3DggWjWuKn7EwyuvvAJEO377a0EFUmYrIpIWamxFRAIomW6E+++/H4CJEycCcNFFFwHRkmpaPDw5ae5GaNGiBRBdKOvQoUP5c/4i2vTp06tbnLoR8hAiLlq1agXA3//+dyDaLqlly5YAzJs3D4g2H/CTIWKmbgQRkbQomcy28iLSPtO98cYbAViwYEFs763MNj9pzmw9v8i0z3Qg2oJnyJAh1S1GmW0ekogLf9GzR48eAFx//fVANHS0SJTZioikRWoz2549ewLR9MpJkyYBMHbsWAC++eYbANavXx/7eyuzzU8pZLbe+PHjy+8feuihABxyyCFAtTaHVGabh1KKiwIpsxURSYvUZrZJUmabn1KKi5122qn8/rRp0wAYPnw4AC+++OLWXq7MNg+lFBcFUmYrIpIWqV9iUSRO3377bfn9du3aJVgTqWuU2YqIBKDGVkQkADW2IiIB5NtnuwyYX4yKpEibpCtQgupCXIBiI1+KiwryGvolIiI1o24EEZEA1NiKiASgxlZEJAA1tiIiAaixFREJQI2tiEgAamxFRAJQYysiEoAaWxGRAP4/VlvJ8uCdn4wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 9 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig2 = plt.figure()\n",
    "for i in range(9):\n",
    "  plt.subplot(3,3,i+1)\n",
    "  plt.tight_layout()\n",
    "  plt.imshow(feature_test[i], cmap='gray', interpolation='none')\n",
    "  plt.title(\"Digit: {}\".format(label_test_predict[i]))\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**预测准确率的局限性:** 无法真实反映模型针对各个分类的预测准确度\n",
    "\n",
    "任务：计算并对比预测模型预测准确率、空准确率\n",
    "\n",
    "空准确率：当模型总是预测比例较高的类别，其预测准确率的数值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**结论:**\n",
    "\n",
    "  分类准确率可以方便的用于衡量模型的整体预测效果，但无法反应细节信息，具体表现在：\n",
    "- 没有体现数据的**实际分布情况**\n",
    "- 没有体现模型**错误预测的类型**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 混淆矩阵\n",
    "\n",
    "又称为误差矩阵，用于衡量分类算法的准确程度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![cat-vs-dog](images/cat-vs-dog.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**名词解释**\n",
    "\n",
    "- **True Positives (TP):** 预测准确、实际为正样本的数量（实际为1，预测为1）\n",
    "- **True Negatives (TN):** 预测准确、实际为负样本的数量（实际为0，预测为0）\n",
    "- **False Positives (FP):** 预测错误、实际为负样本的数量（实际为0，预测为1）\n",
    "- **False Negatives (FN):** 预测错误、实际为正样本的数量（实际为1，预测为0）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Small confusion matrix](images/09_confusion_matrix_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Large confusion matrix](images/09_confusion_matrix_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 混淆矩阵指标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**准确率:** 整体样本中，预测正确样本数的比例\n",
    "- Accuracy = (TP + TN)/(TP + TN + FP + FN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**精确率:** 预测结果为正的样本中，预测正确的比例\n",
    "- Precision = TP/(TP + FP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**F1分数:** 综合Precision和Recall的一个判断指标\n",
    "- F1 Score = 2*Precision X Recall/(Precision + Recall)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![confusion matrix_metrics](images/09_confusion_matrix_3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**结论:**\n",
    "\n",
    "- 分类任务中，相比单一的预测准确率，混淆矩阵提供了**更全面的模型评估信息**\n",
    "- 通过混淆矩阵，我们可以计算出**多样的模型表现衡量指标**，从而更好地选择模型\n",
    "\n",
    "**哪个衡量指标更关键?**\n",
    "\n",
    "- 衡量指标的选择取决于**应用场景**\n",
    "- **垃圾邮件检测** (正样本为 \"垃圾邮件\"): 希望普通邮件（负样本）不要被判断为垃圾邮件（正样本），需要关注**精确率**，希望判断为垃圾邮件的样本都是判断正确的；还需要关注**召回率**，希望所有的垃圾邮件尽可能被判断出来)\n",
    "- **异常交易检测** (正样本为 \"异常交易\"): 希望所有的异常交易都被检测到，即判断为正常的交易中尽可能不存在异常交易，需要关注**特异度**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
